System and method for single-frame based super resolution interpolation for digital cameras

ABSTRACT

A digital camera system for super resolution image processing is provided. The digital camera system includes a resolution enhancement module configured to receive at least a portion of an image, to increase the resolution of the received image, and to output a resolution enhanced image and an edge extraction module configured to receive the resolution enhanced image, to extract at least one edge of the resolution enhanced image, and to output the extracted at least one edge of the resolution enhanced image, the at least one edge being a set of contiguous pixels where an abrupt change in pixel values occur. The digital camera system also includes an edge enhancement module configured to receive the resolution enhanced image and the extracted at least one edge, and to combine the extracted at least one edge or a derivation of the extracted at least one edge with the resolution enhanced image.

BACKGROUND

1. Technical Field

The present invention relates to systems and methods for single-framebased super resolution interpolation for digital cameras.

2. Discussion

Digital cameras use various systems to enable photographers to captureimages of objects at long distances. Optical zoom systems use one ormore zoom lenses that may be adjusted to narrow the field of visibleradiation incident on the photo-detectors of the digital camera. Thenarrower field of visible radiation incident on the photo-detectorsmagnifies the captured image, albeit with a narrower field of view,without the introduction of significant image aberrations. In contrast,digital zoom systems process the image, or a subset of the image, toincrease its resolution to create an effect similar to optical zoom(i.e., a magnified narrower field of view). Digital zoom systems,however, generally produce significant undesirable image aberrationsrelative to optical zoom systems. For example, digital zoom systems mayintroduce aliasing (i.e., jagged diagonal edges), blurring, and/orhaloing into the image. The aberrations introduced during digital zoomprocess occur primarily at or around the edges of objects in the image.

SUMMARY OF INVENTION

In accordance with at least one aspect of the embodiments disclosedherein, it is recognized that the standard approach to digital zoomsystems produces significant image aberrations including, but notlimited to, aliasing, blurring, and/or haloing. To minimize theintroduction of these image aberrations through digital zoom, systemsand methods for single-frame based super resolution interpolation fordigital cameras are provided. The systems and methods for superresolution digitally zooms images by increasing the resolution of theimage, or any portion of the image, in addition to extracting andenhancing the edges of objects in the image. The edges of the image areenhanced through a plurality of edge-preserving filters. These filtersenable the extraction and enhancement of the edges in the image toincrease the detail of digitally zoomed images.

According to one aspect, a digital camera system for super resolutionimage processing is provided. The digital camera system comprises aresolution enhancement module configured to receive at least a portionof an image, to increase the resolution of the received image, and tooutput a resolution enhanced image, an edge extraction module configuredto receive the resolution enhanced image, to extract at least one edgeof the resolution enhanced image, and to output the extracted at leastone edge of the resolution enhanced image, the at least one edge being aset of contiguous pixels where an abrupt change in pixel values occur,and an edge enhancement module configured to receive the resolutionenhanced image and the extracted at least one edge, and to combine theextracted at least one edge or a derivation of the extracted at leastone edge with the resolution enhanced image.

According to one embodiment, the resolution enhancement module isfurther configured to increase the resolution of the received image atleast in part by interpolating the received image. According to oneembodiment, the resolution enhancement module is further configured tointerpolate the received image consistent with a bicubic interpolationprocess. According to one embodiment, the edge extraction module isfurther configured to filter the resolution enhanced image through afirst edge-preserving filter having a first strength and a secondedge-preserving filter having a second strength. According to oneembodiment, the strength of the first edge-preserving filter isconfigurable and wherein the edge extraction module is furtherconfigured to decrease the strength of the first edge-preserving filterto increase a level of image detail or increase the strength of thefirst edge-preserving filter to decrease the level of image detail.According to one embodiment, the first strength is less than the secondstrength and wherein the edge extraction module is further configured tosubtract the resolution enhanced image filtered through the secondedge-preserving filter from the resolution enhanced image filteredthrough the first edge-preserving filter. According to one embodiment,the edge enhancement module is further configured to add the extractedat least one edge to the resolution enhanced image.

According to one embodiment, the resolution enhancement module isfurther configured to divide the received image into a plurality ofsubsections. According to one embodiment, the resolution enhancementmodule is further configured to increase the resolution of each of thesubsections of the received image and wherein the edge extractingcomponent is further configured to extract at least one edge of eachsubsection of the resolution enhanced image. According to oneembodiment, the digital camera system further comprises a white noisemodule configured to add white noise to the resolution enhanced image.

According to one aspect, a method of super resolution image processingin a digital camera is provided. The method comprises increasing theresolution of a received image to obtain a resolution enhanced image,extracting at least one edge of the resolution enhanced image, the atleast one edge being a set of contiguous pixels where an abrupt changein pixel values occur, and enhancing the at least one edge of theresolution enhanced image by combining the extracted at least one edgeor a derivation of the extracted at least one edge with the resolutionenhanced image.

According to one embodiment, the method further comprises increasing theresolution of the received image includes interpolating the receivedimage. According to one embodiment, interpolating the received imageincludes interpolating the received image consistent with a bicubicinterpolation process. According to one embodiment, extracting the atleast one edge includes filtering the resolution enhanced image througha first edge-preserving filter having a first strength and a secondedge-preserving filter having a second strength. According to oneembodiment, the strength of the first edge-preserving filter isconfigurable and extracting the at least one edge includes increasingthe strength of the first edge-preserving filter to decrease a level ofimage detail or decreasing the strength of the first edge-preservingfilter to increase a level of image detail. According to one embodiment,the first strength is less than the second strength and whereinextracting the at least one edge further includes subtracting theresolution enhanced image filtered through the second edge-preservingfilter from the resolution enhanced image filtered through the firstedge-preserving filter. According to one embodiment, enhancing the atleast one edge further includes adding the extracted at least one edgeto the resolution enhanced image.

According to one embodiment, increasing the resolution of the receivedimage includes dividing the received image into a plurality of receivedimage subsections. According to one embodiment, increasing theresolution of the received image further includes increasing theresolution of each of the subsections of the received image to obtain aplurality of resolution enhanced image subsections and wherein enhancingthe at least one edge includes enhancing at least one edge in each ofthe resolution enhanced image subsections. According to one embodiment,the method further comprises adding white noise to the resolutionenhanced image.

Still other aspects, embodiments and advantages of these exemplaryaspects and embodiments, are discussed in detail below. Moreover, it isto be understood that both the foregoing information and the followingdetailed description are merely illustrative examples of various aspectsand embodiments, and are intended to provide an overview or frameworkfor understanding the nature and character of the claimed aspects andembodiments. Any embodiment disclosed herein may be combined with anyother embodiment. References to “an embodiment,” “an example,” “someembodiments,” “some examples,” “an alternate embodiment,” “variousembodiments,” “one embodiment,” “at least one embodiment,” “this andother embodiments” or the like are not necessarily mutually exclusiveand are intended to indicate that a particular feature, structure, orcharacteristic described in connection with the embodiment may beincluded in at least one embodiment. The appearances of such termsherein are not necessarily all referring to the same embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of at least one example are discussed below withreference to the accompanying figures, which are not intended to bedrawn to scale. The figures are included to provide an illustration anda further understanding of the various aspects and examples, and areincorporated in and constitute a part of this specification, but are notintended as a definition of the limits of the embodiments disclosedherein. The drawings, together with the remainder of the specification,serve to explain principles and operations of the described and claimedaspects and examples. In the figures, each identical or nearly identicalcomponent that is illustrated in various figures is represented by alike numeral. For purposes of clarity, not every component may belabeled in every figure. In the figures:

FIG. 1 illustrates a functional block diagram of a super resolutionimage processing system in accordance with the present invention;

FIG. 2 illustrates an image processing system in which super resolutionimage processing may be performed;

FIG. 3 illustrates a more detailed functional block diagram of a superresolution image processing system for super resolution image processingin accordance with an embodiment of the present invention;

FIG. 4 illustrates a more detailed functional block diagram of a superresolution image processing system for super resolution image processingin accordance with another embodiment of the present invention;

FIG. 5 illustrates a more detailed functional block diagram of a superresolution image processing system for super resolution image processingin accordance with yet another embodiment of the present invention; and

FIG. 6 illustrates example diagrams of an edge smoothing process appliedto a pixel matrix.

DETAILED DESCRIPTION

It is to be appreciated that examples of the methods and apparatusesdiscussed herein are not limited in application to the details ofconstruction and the arrangement of components set forth in thefollowing description or illustrated in the accompanying drawings. Themethods and apparatuses are capable of implementation in other examplesand of being practiced or of being carried out in various ways. Examplesof specific implementations are provided herein for illustrativepurposes only and are not intended to be limiting. In particular, acts,elements and features discussed in connection with any one or moreexamples are not intended to be excluded from a similar role in anyother examples.

Also, the phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. Any references toexamples or elements or acts of the systems and methods herein referredto in the singular may also embrace examples including a plurality ofthese elements, and any references in plural to any example or elementor act herein may also embrace examples including only a single element.References in the singular or plural form are not intended to limit thepresently disclosed systems or methods, their components, acts, orelements. The use herein of “including,” “comprising,” “having,”“containing,” “involving,” and variations thereof is meant to encompassthe items listed thereafter and equivalents thereof as well asadditional items. References to “or” may be construed as inclusive sothat any terms described using “or” may indicate any of a single, morethan one, and all of the described terms.

Embodiments of the present invention relate to single-frame based superresolution interpolation for digital cameras. It should be appreciatedthat the term “digital camera” used herein includes, but is not limitedto, dedicated cameras as well as camera functionality performed by anyelectronic device (e.g., mobile phones, personal digital assistants,etc.). In addition, the methods and systems described herein may beapplied to a plurality of images arranged in a time sequence (e.g., avideo stream). The use herein of the term “module” is interchangeablewith the term “block” and is meant to include implementations ofprocesses and/or functions in software, hardware, or a combination ofsoftware and hardware. In one embodiment, the single-frame based superresolution interpolation system and method digitally zoom an image, orany portion of the image, without producing significant imageaberrations in the output digitally zoomed image. Digitally zooming animage causes significant aberrations particularly to the edges ofobjects in the image. For example, the edges in digitally zoomed imagesmay appear jagged from aliasing. Accordingly, in some embodiments, thesystems and methods provided improve the quality of the edges in thesuper resolution process to produce a high quality digitally zoomedimage.

Super Resolution Image Processing System

FIG. 1 illustrates an example system for super resolution imageprocessing. As shown, the super resolution image processing system 100of FIG. 1 includes a super resolution engine 110 that receives an image102 and performs image processing on the received image to produce adigitally zoomed image 104. The super resolution engine 110 includesmultiple functional blocks or modules including an image regionselection block 112 that selects at least a portion of the image 102 andprovides an image region 106, a resolution enhancement block 114 thatenhances the resolution of the portion of the image and provides aresolution enhanced image region 108, and an edge enhancement block 116that enhances edges in the enhanced image region and provides thedigitally zoomed image 104.

The input to the super resolution engine 110 is an image 102. In oneembodiment, the image comprises a plurality of pixels represented by oneor more two dimensional matrices. Each element in the matricesrepresents information regarding a specific pixel location in the image.For example, the location (10, 10) in a first, second, and third twodimensional matrix may represent data associated with a pixel at the(10, 10) location in the image. Each of the one or more matrices mayrepresent a set of intensities associated with the image in a givendimension. In one embodiment, each pixel in the image is represented bya three-dimensional vector. In this embodiment, the image is representedby a first two dimensional matrix for the first dimension, a second twodimensional matrix for the second dimension, and a third two dimensionalmatrix for the third dimension. It is appreciated that each pixel may berepresented by any number of dimensions or any number of matrices.

In some embodiments, the image is represented by three two dimensionalmatrices consistent with a YUV color space. The YUV color space iscomposed of three distinct components Y, U, and V where each twodimensional matrix represents a specific component of the YUV space. TheY component is the luminance component that relates to the brightness ofthe image. The U and V components are chrominance components that relateto the specific color of the image. Each pixel in an image isrepresented by a vector in the YUV color space (i.e., some combinationof the Y, U, and V components). In some embodiments, the image isrepresented by three two dimensional matrices consistent with a RGBcolor space. The RGB color space is also composed of three distinctcomponents R, G, and B where each two dimensional matrix represents aspecific component of the RGB space. All three of the distinctcomponents (i.e., R, G, and B components) are all chrominance componentsthat relate to the specific color of the image. It is appreciated thatthe image may be represented in any color space and is not limited tothe YUV or RGB color spaces.

The image 102 is input into the super resolution engine 110. The superresolution engine 110 executes a variety of processes to output adigitally zoomed image 104. In one embodiment, the digitally zoomedimage is a magnified image region 106. The image region 106 may includethe entire image or any subset of the image 102. It is appreciated thatthe processes performed by the super resolution engine 110 do not needto be performed on all of the dimensions of each pixel of the imageregion 106. In some embodiments, the pixels in the image region 106 arerepresented by a vector in the YUV color space. In these embodiments,the processes of the super resolution engine 110 may be performed on anycombination dimensions in the YUV color space. For example, in oneembodiment the processes of the super resolution engine may be performedonly on the Y component (i.e., the luminance component) of the imageregion 106 pixels. In this embodiment, interpolation processes (e.g.,bicubic interpolation) may be performed on the U and V components (i.e.,the chrominance components) to obtain a digitally zoomed image 104.

The various processes performed by the super resolution engine 110 togenerate the digitally zoomed image 104 include the processes performedby the image region selection block 112, the resolution enhancementblock 114, and the edge enhancement block 116.

In various embodiments, the image region selection block 112 crops theimage as appropriate to select an image region 106 of the input image102 to be digitally zoomed. It is appreciated that the image regionselection block 112 is an optional block. For example, the superresolution engine 110 may digitally zoom the entire image 102. In otherembodiments, the super resolution engine 110 self-selects the entireimage 102 or any subset of the image 102 to digitally zoom.

In some embodiments, the resolution enhancement block 114 increases theresolution of the image region 106 to produce an enhanced resolutionimage region 108. In one embodiment, the resolution of the image region106 is increased through interpolation of the image region 106.Interpolation is a method of creating new data points between a set ofknown discrete data points. In this embodiment, the known data pointsare the known pixel values (e.g., the Y, U, and/or V channel intensityvalues for a given pixel). The new data points, in this embodiment, arethe pixels created through interpolation. It is appreciated that theinterpolation may be preceded by one or more acts to smooth the edges ofthe image region 106 to improve the quality of the edges in theresolution enhanced image region 108. The edge smoothing sharpens theimage by shortening the transitions between color and/or brightnesschanges (i.e., object edges). A visual representation of an edgesmoothing process is illustrated in the edge smoothing process diagrams600 with reference to FIG. 6. The super resolution engine 110 mayproceed to the edge enhancement block 116.

In one embodiment, the edge enhancement block 116 enhances the edges ofobjects in the image. The interpolation performed in the resolutionenhancement block 114 may create a smoothed image that lacks sharpdetails. Accordingly, the edge enhancement block 116 is capable ofextracting and enhancing the edges of the resolution enhanced imageregion 108. In one embodiment, the edge enhancement block 116 extractsthe edges of the resolution enhanced image region 108. The extractededges may then be added to the resolution enhanced image region 108 toincrease the edge detail.

It is appreciated that any combination of the super resolution imageprocessing functional blocks (e.g., the image region selection block112, the resolution enhancement block 114, and the edge enhancementblock 116) may perform operations in parallel. In one embodiment, theinput image 102 may be divided into a plurality of subsections. Eachsubsection of the image 102 may be processed individually in parallel.The processed subsections may be combined to form the digitally zoomedimage 104. Implementing the super resolution image processing inparallel increases the speed in which the digitally zoomed image 104 canbe provided.

It is appreciated that the super resolution engine 110 may take avariety of forms dependent upon the specific application. In someembodiments, the super resolution engine 110 comprises a series ofinstructions performed by an image processing system. In otherembodiments, one or more of the blocks or modules 112, 114, and 116 maybe implemented in hardware, such as an application-specific integratedcircuit (ASIC), system on a chip (SoC), state machine, or a dedicatedprocessor.

Example Image Processing System

FIG. 2 illustrates an example image processing system 200 in which superresolution processing can be performed. As shown, the image processingsystem 200 of FIG. 2 includes a digital camera 202 that is configured toprovide a digital image of a scene 218. The digital camera 202 includesan image processor 204, an analog-to-digital converter 206, a memory208, an interface 210, storage 212, an image sensor 214, and a lens 216.

As illustrated in FIG. 2, the digital camera 202 includes the imageprocessor 204 to implement at least some of the aspects, functions andprocesses disclosed herein. The image processor 204 performs a series ofinstructions that result in manipulated data. The image processor 204may be any type of processor, multiprocessor or controller. Someexemplary processors include processors with ARM11 or ARM9architectures. The image processor 204 is connected to other systemcomponents, including one or more memory devices 208.

The memory 208 stores programs and data during operation of the digitalcamera 202. Thus, the memory 208 may be a relatively high performance,volatile, random access memory such as a dynamic random access memory(“DRAM”) or static memory (“SRAM”). However, the memory 208 may includeany device for storing data, such as a flash memory or othernon-volatile storage devices. Various examples may organize the memory208 into particularized and, in some cases, unique structures to performthe functions disclosed herein. These data structures may be sized andorganized to store values for particular data and types of data.

The data storage element 212 includes a writeable nonvolatile, ornon-transitory, data storage medium in which instructions are storedthat define a program or other object that is executed by the imageprocessor 204. The data storage element 212 also may include informationthat is recorded, on or in, the medium, and that is processed by theimage processor 204 during execution of the program. More specifically,the information may be stored in one or more data structuresspecifically configured to conserve storage space or increase dataexchange performance. The instructions may be persistently stored asencoded signals, and the instructions may cause the image processor 204to perform any of the functions described herein. The medium may, forexample, be optical disk, magnetic disk or flash memory, among others.In operation, the image processor 204 or some other controller causesdata to be read from the nonvolatile recording medium into anothermemory, such as the memory 208, that allows for faster access to theinformation by the image processor 204 than does the storage mediumincluded in the data storage element 212. The memory may be located inthe data storage element 212 or in the memory 208, however, the imageprocessor 204 manipulates the data within the memory, and then copiesthe data to the storage medium associated with the data storage element212 after processing is completed. A variety of components may managedata movement between the storage medium and other memory elements andexamples are not limited to particular data management components.Further, examples are not limited to a particular memory system or datastorage system.

The digital camera 202 also includes one or more interface devices 210such as input devices, output devices and combination input/outputdevices. Interface devices may receive input or provide output. Moreparticularly, output devices may render information for externalpresentation. Input devices may accept information from externalsources. Examples of interface devices include microphones, touchscreens, display screens, speakers, buttons, etc. Interface devicesallow the digital camera 202 to exchange information and to communicatewith external entities, such as users and other systems.

Although the digital camera 202 is shown by way of example as one typeof digital camera upon which various aspects and functions may bepracticed, aspects and functions are not limited to being implemented onthe digital camera 202 as shown in FIG. 2. Various aspects and functionsmay be practiced on digital cameras having a different architectures orcomponents than that shown in FIG. 2. For instance, the digital camera202 may include specially programmed, special-purpose hardware, such asan application-specific integrated circuit (“ASIC”) or a system on achip (“SoC”) tailored to perform a particular operation disclosedherein. It is appreciated that the digital camera 202 may beincorporated into another electronic device (e.g., mobile phone,personal digital assistant etc.) and is not limited to dedicated digitalcameras.

The lens 216 includes one or more lenses that focus the visibleradiation on the image sensor 214. It is appreciated that the lens 216is not limited to a single physical lens as illustrated in FIG. 2. Insome embodiments, the lens 216 includes a plurality of zoom lenses thatenable optical zoom. Optical zoom may be accomplished by narrowing thefield of view of the visible radiation incident on the image sensor 214.

The image sensor 214 may include a two dimensional area of sensors(e.g., photo-detectors) that are sensitive to light. In someembodiments, the photo-detectors of the image sensor 214, in someembodiments, can detect the intensity of the visible radiation in one oftwo or more individual color and/or brightness components. For example,the output of the photo-detectors may include values consistent with aYUV or RGB color space. It is appreciated that other color spaces may beemployed by the image sensor 214 to represent the captured image.

In various embodiments, the image sensor 214 outputs an analog signalproportional to the intensity and/or color of visible radiation strikingthe photo-detectors of the image sensor 214. The analog signal output bythe image sensor 214 may be converted to digital data by theanalog-to-digital converter 206 for processing by the image processor204. In some embodiments, the functionality of the analog-to-digitalconverter 206 is integrated with the image sensor 214. The imageprocessor 204 may perform variety of processes to the captured image.These processes may include, but are not limited to, one or more superresolution processes to digitally zoom the captured image.

Super Resolution Image Processing Implementations

FIG. 3 illustrates a more detailed functional block diagram of the superresolution image processing system 100 including an image regionselection block 112, a resolution enhancement block 114, and an edgeenhancement block 116 in accordance with FIG. 1. As shown in FIG. 3, thesuper resolution image processing system 300 of FIG. 3 includes a cropimage block 302, an edge smoothing block 304, an interpolate block 306,an edge extraction block 307, a fine edge-preserving filter block 308, acourse edge-preserving filter block 310, an edge processing functionblock 312, a white noise block 314, a difference block 316, and a sumblock 318. It is appreciated that any block drawn in dashed lines is anoptional block. In addition, large boxes drawn in dotted lines show therelation between the super resolution image processing system 100including the image region selection block 112, the resolutionenhancement block 114, and the edge enhancement block 116 described withreference to FIG. 1 and the functional blocks shown in FIG. 3.

The crop image block 302 illustrates one possible implementation of theimage region selection block 112. The crop image block 302 crops theimage as appropriate to feed into the later functional blocks of thesuper resolution image processing system 300. In one embodiment, thecrop image block 302 may crop the image responsive to input receivedthrough an interface from an external entity (e.g., interface 210 of thedigital camera 202). In another embodiment, the crop image block 302self-selects an image region 106 and crops the image 102 as appropriate.It is appreciated that the crop image block 302 is an optional block.

The optional image region selection block 112 is followed by theresolution enhancement block 114. The edge smoothing block 304 and theinterpolate block 306 illustrate one possible implementation of theresolution enhancement block 114. In one embodiment, the edge smoothingblock 304 smoothes the edges of objects in the image region 106. Theedge of an object in an image may be characterized by a set ofcontiguous pixels where an abrupt change of intensity values occurs.

A visual representation of an edge smoothing process is illustrated bythe edge smoothing process diagrams 600 with reference to FIG. 6. Asshown in FIG. 6, the edge smoothing process diagrams 600 of FIG. 6includes an object 608 within the field of view of a pixel matrix 610that proceeds through a pre-captured image state 602, a captured imagestate 604, and an edge smoothed image state 606.

The pre-captured image state 602 illustrates an object 608 in the fieldof view of a pixel matrix 610. The edges of the object 608 cross througha plurality of pixels in the pixel matrix. The captured image state 604of the pixel matrix 610 illustrates the effect of capturing an image ofan object with edges crossing through multiple pixels. The third row ofpixels from the top illustrates the edge blurring effect that occurswhen an image is captured by a digital camera (e.g., digital camera202). The pixels in the third row from the top get darker moving fromleft to right proportional to the area of the pixel that was covered bythe object. The edge smoothing process that occurs between the imagecaptured state 604 and the edge smoothed image state 606 sharpens theimage by eliminating some or all of the transition pixels. In thisillustration, the elimination of transition pixels is shown in the thirdrow of the pixel matrix 610 in the edge smoothed image state 606. Thedegree of the transition (i.e., the amount of transition pixelsremaining in the image region) may be controlled with a strengthparameter associated with the edge smoothing filter applied.

Referring back to FIG. 3, the edge smoothing block 304 may be anypertinent edge smoothing or edge-preserving filter. In one embodiment,the edge smoothing filter is the edge preserving noise reduction filterdescribed in co-pending U.S. application Ser. No. 13/160,775, titledADVANCED NOISE REDUCTION IN DIGITAL CAMERAS, filed Jun. 15, 2011(hereinafter the “'775 application”), which is incorporated by referenceherein in its entirety. It should be appreciated that embodiments of thepresent invention are not limited to any particular type of edgesmoothing or edge preserving filter, as various types of filters may beused.

The edge smoothing block is followed by the interpolate block 306. Theinterpolate block may compute a new pixel matrix at a higher resolutionbased on the input pixel matrix. In one embodiment, the interpolationblock analyzes a closest set of pixels from the input image pixel matrixto determine corresponding new pixel values in the new pixel matrix. Thespecific process that is used to perform the interpolation may beconsistent with a variety of methods including, but not limited to,bicubic interpolation, bilinear interpolation, nearest neighborinterpolation, spline interpolation, or sinc interpolation.

The resolution enhancement block 114 is followed by the edge enhancementblock 116. The edge enhancement block 116, in some embodiments, isillustrated by the edge extraction block 307, the fine edge-preservingfilter 308, the course edge-preserving filter 310, the edge processingfunction block 312, the difference block 316, and the sum block 318. Inone embodiment, the fine edge-preserving filter 308 and the courseedge-preserving filter 310 are edge sensitive filters of differentstrengths. In this embodiment, the fine edge-preserving filter 308 is aweaker edge filter than the course edge-preserving filter 310. It isappreciated that the fine edge-preserving filter 308 may or may not bethe same type of edge-preserving filter as the course edge-preservingfilter 310. The output of the course edge-preserving filter 310 issubtracted from the output of the fine edge preserving filter 308 indifference block 316. The output of the difference block 316 containsonly the edges of the resolution enhanced image region 108. Thecombination of the fine edge-preserving filter 308, the courseedge-preserving filter 310 and the difference block 318 may thus formthe edge extraction block 307. It is appreciated that the specificmethod to obtain the edges of the resolution enhanced image region 108may be altered. For example, a plurality of edge-preserving filters ofvarious strengths may be combined in a linear fashion to generate asimilar resultant output (i.e., the edges of the image). In addition, asingle filter that targets the edges of the images may also be used. Inother embodiments, the fine edge-preserving filter 308 and the courseedge-preserving filter 310 can be the edge preserving noise reductionfilters described in the '775 application. It is appreciated that anycombination of the edge-preserving filters may be selectable, forexample, based on input or preferences received from a user of thesystem.

The image edges from the difference block 316 may then be processed bythe edge processing function block 312. The edge processing functionblock may be any function, linear or otherwise, that enhances theresolution enhanced image 106 edges. The edge processing function blockmay be implemented, for example, with a lookup table (LUT). In oneembodiment, the edge processing function block 312 is a multiplier thatamplifies or attenuates the edges. In this embodiment, the edgeprocessing function block 312 may receive input from a user through auser interface. For example, the user interface may have configurablesettings regarding the level of detail in the digitally zoomed image104. The edge processing function block 312 may increase the multiplierresponsive to a higher detail level setting or decrease the multiplierresponsive to a lower detail level setting. It is appreciated that theedge processing function block 312 is an optional function. The outputof the difference block 312 may be directly connected to the summationblock 318.

The summation block 318 combines the resolution enhanced image region106 with the edges of the resolution enhanced image region 106 to form adigitally zoomed image 104. The addition of the edges of the resolutionenhanced image region 106 to the resolution enhanced image region 106increases the detail of the edges that may have undergone some level ofdistortion in the resolution enhancement block 114.

The summation block 318 may also combine white noise from the whitenoise block 314 with the digitally zoomed image 104. The addition ofwhite noise to the digitally zoomed image 104 gives the appearance ofhigh resolution noise in the digitally zoomed image 104 rather thanamplified low resolution noise. The intensity of white noise added mayvary. For example, in one embodiment the amount of white noise added tothe image is computed responsive to the camera settings when the imagewas taken (e.g., ISO, aperture, or shutter speed). It is appreciatedthat the addition of white noise, and consequently the white noise block314, is optional. In addition, the white noise from the white noisemodule may be combined with the resolution enhanced image region 106.

Having described at least one implementation of the super resolutionimage processing system, it is appreciated that the specificimplementations described may be altered to generate the same result(i.e., a digitally zoomed image without significant image aberrations).Other example implementations of the super resolution image processingsystems are illustrated by functional block diagrams of super resolutionimage processing systems 400 and 500 with reference to FIGS. 4 and 5respectively.

FIG. 4 illustrates an alternative functional block diagram of the superresolution image processing system 100 including an image regionselection block 112, a resolution enhancement block 114, and an edgeenhancement block 116 in accordance with FIG. 1. As shown in FIG. 4, thesuper resolution image processing system 400 of FIG. 4 includes a cropimage block 302, edge smoothing blocks 304 and 404, an interpolationblock 306, an edge extraction block 307, a fine edge-preserving filterblock 308, a course edge-preserving filter block 310, an edge processingfunction block 312, a white noise block 314, a difference block 316, anda sum block 318. It is appreciated that any block drawn in dashed linesis an optional block. In addition, large boxes drawn in dotted linesshow the relation between the processes described with reference to FIG.1 and the functional blocks shown in FIG. 4.

The image region selection block 112 and the edge enhancement block 116implemented by the crop image block 302, the edge extraction block 307,the fine edge-preserving filter 308, the course edge-preserving filter310, the edge processing function block 312, the difference block 316,and the sum block 318 are similar to that described with reference FIG.3 above. The resolution enhancement block 114 that is implemented in theembodiment depicted in FIG. 4 by the edge smoothing blocks 304 and 404in addition to the interpolate block 306 is a variation of theresolution enhancement block 116 described with reference to FIG. 3. Theadditional edge smoothing block 404 improves the quality of the edges inthe resolution enhanced image 106 and consequently the final digitallyzoomed image 104. The additional cost of the additional edge smoothingblock 404 is the additional computation required to perform theadditional edge smoothing. It is appreciated that the filter used in theedge smoothing block 404 may, or may not, be constructed to be identicalto the edge filter in the edge smoothing block 304. In addition, thesummation block 318 may also combine white noise from the white noiseblock 314 with the digitally zoomed image 104 as described withreference to FIG. 3.

FIG. 5 illustrates a further alternative functional block diagram of thesuper resolution image processing system 100 including an image regionselection block 112, are solution enhancement block 114, and an edgeenhancement block 116 in accordance with FIG. 1. As shown in FIG. 5, thesuper resolution image processing system 500 of FIG. 5 includes a cropimage block 302, edge smoothing blocks 304 and 504, interpolation blocks306 and 406, edge extraction blocks 307 and 507, fine edge-preservingfilter blocks 308 and 508, course edge-preserving filter blocks 310 and410, edge processing function blocks 312 and 512, white noise block 514,difference blocks 316 and 516, and sum blocks 318 and 518. It isappreciated that any block drawn in dashed lines is an optional block.In addition, large boxes drawn in dotted lines show the relation betweenthe processes described with reference to FIG. 1 and the functionalblocks shown in FIG. 5.

The image region selection block 112, resolution enhancement block 114,and the edge enhancement block 116 are similar to that described withreference FIG. 3 above. The implementation of the super resolution imageprocessing system in FIG. 5 shows multiple iterations of the resolutionenhancement block 114 and the edge enhancement block 116. Multipleiterations of the resolution enhancement block 114 and the edgeenhancement block 116 processes increase the detail in the digitallyzoomed image 104 at the cost of additional computation. It isappreciated that the resolution enhancement block 114 and the edgeenhancement block 116 may be repeated any number of times to achieve thedesired level of detail in the digitally zoomed image 104. In oneembodiment, the image 102 may need to be digitally zoomed substantially.In this embodiment, the first iteration of the resolution enhancementblock 114 and the edge enhancement block 116 may only digitally zoom theimage 102 a fraction of the required total digital zoom. The seconditeration of the resolution enhancement block 114 and the edgeenhancement block 116 may increase the resolution to the desired finaldigitally zoomed image 104. It should be appreciated that an additionaledge smoothing block (e.g., edge smoothing block 404 in FIG. 4) can beadded to the first iteration, the second iteration, or both. Inaddition, the summation block 518 may also combine white noise from thewhite noise block 514 with the digitally zoomed image 104 after bothiterations of the super resolution processes.

It is appreciated that the super resolution image processing system 100including the image region selection block 112, the resolutionenhancement block 114, and the edge enhancement block 116 may beperformed on any combination of the chrominance and/or luminancechannels of the image 102. For example, in one embodiment the image 102is in the YUV color space and the super resolution image processingsystem 100 including the image region selection block 112, theresolution enhancement block 114, and the edge enhancement block 116 areonly applied to the luminance channel (i.e., Y channel) of the image102. In this embodiment, the U and V channels undergo only interpolation(e.g., bicubic interpolation). This embodiment is of lower computationalcomplexity when compared to performing the super resolution processes onall three channels. The loss of quality is minimal, in this embodiment,because the human eye is substantially more sensitive to brightness thancolor. The sensitivity of the photoreceptors that perceive brightness(i.e., rods of the human eye) are more numerous and more sensitive thanphotoreceptors that perceive color (i.e., cones of the human eye).

Having thus described several aspects of at least one example, it is tobe appreciated various alterations, modification, and improvements willreadily occur to those skilled in the art. For example, it should beappreciated that specific functional blocks in FIGS. 3-5 may be altered.For example depending on the requirements, some implementations may usea single filter to isolate the edges of the image rather than computingthe difference between two edge-preserving filters of differentstrengths. Moreover, it should be appreciated that in certainenvironments any linear combination of edge-preserving filters may beused to fine-tune edge enhancement process. Such alterations,modifications, and improvements are intended to be part of thisdisclosure, and are intended to be within the scope of the embodimentsdisclosed herein. Accordingly, the foregoing description and drawingsare by way of example only.

What is claimed is:
 1. A digital camera system for super resolutionimage processing, comprising: a resolution enhancement processorconfigured to receive at least a portion of an image, to smooth at leastone edge of the portion, to increase the resolution of the smoothedportion via interpolation, and to output a resolution enhanced image; anedge extraction processor configured to receive the resolution enhancedimage, to extract at least one edge of the resolution enhanced image,and to output the extracted at least one edge of the resolution enhancedimage, the at least one edge being a set of contiguous pixels where anabrupt change in pixel values occur; and an edge enhancement processorconfigured to receive the resolution enhanced image and the extracted atleast one edge, and to combine the extracted at least one edge or aderivation of the extracted at least one edge with the resolutionenhanced image.
 2. The digital camera system of claim 1, wherein theresolution enhancement processor is further configured to interpolatethe received image consistent with a bicubic interpolation process. 3.The digital camera system of claim 1, wherein the edge extractionprocessor is further configured to filter the resolution enhanced imagethrough a first edge-preserving filter having a first strength and asecond edge-preserving filter having a second strength.
 4. The digitalcamera system of claim 3, wherein the strength of the firstedge-preserving filter is configurable and wherein the edge extractionprocessor is further configured to decrease the strength of the firstedge-preserving filter to increase a level of image detail or increasethe strength of the first edge-preserving filter to decrease the levelof image detail.
 5. The digital camera system of claim 3, wherein thefirst strength is less than the second strength and wherein the edgeextraction processor is further configured to subtract the resolutionenhanced image filtered through the second edge-preserving filter fromthe resolution enhanced image filtered through the first edge-preservingfilter.
 6. The digital camera system of claim 5, wherein the edgeenhancement processor is further configured to add the extracted atleast one edge to the resolution enhanced image.
 7. The digital camerasystem of claim 1, wherein the resolution enhancement processor isfurther configured to divide the received image into a plurality ofsubsections.
 8. The digital camera system of claim 7, wherein theresolution enhancement processor is further configured to increase theresolution of each of the subsections of the received image and whereinthe edge extracting component is further configured to extract at leastone edge of each subsection of the resolution enhanced image.
 9. Thedigital camera system of claim 1, wherein the digital camera systemfurther comprises a white noise processor configured to add white noiseto the resolution enhanced image.
 10. A method of super resolution imageprocessing in a digital camera, comprising: receiving at least a portionof an image; smoothing at least one edge of the portion; increasing theresolution of the smoothed portion via interpolation to obtain aresolution enhanced image; extracting at least one edge of theresolution enhanced image, the at least one edge being a set ofcontiguous pixels where an abrupt change in pixel values occur; andenhancing the at least one edge of the resolution enhanced image bycombining the extracted at least one edge or a derivation of theextracted at least one edge with the resolution enhanced image.
 11. Themethod of claim 10, wherein interpolating the received image includesinterpolating the received image consistent with a bicubic interpolationprocess.
 12. The method of claim 10, wherein extracting the at least oneedge includes filtering the resolution enhanced image through a firstedge-preserving filter having a first strength and a secondedge-preserving filter having a second strength.
 13. The method of claim12, wherein the strength of the first edge-preserving filter isconfigurable and extracting the at least one edge includes increasingthe strength of the first edge-preserving filter to decrease a level ofimage detail or decreasing the strength of the first edge-preservingfilter to increase a level of image detail.
 14. The method of claim 12,wherein the first strength is less than the second strength and whereinextracting the at least one edge further includes subtracting theresolution enhanced image filtered through the second edge-preservingfilter from the resolution enhanced image filtered through the firstedge-preserving filter.
 15. The method of claim 14, wherein enhancingthe at least one edge further includes adding the extracted at least oneedge to the resolution enhanced image.
 16. The method of claim 10,wherein increasing the resolution of the received image includesdividing the received image into a plurality of received imagesubsections.
 17. The method of claim 16, wherein increasing theresolution of the received image further includes increasing theresolution of each of the subsections of the received image to obtain aplurality of resolution enhanced image subsections and wherein enhancingthe at least one edge includes enhancing at least one edge in each ofthe resolution enhanced image subsections.
 18. The method of claim 10,further comprising adding white noise to the resolution enhanced image.